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Abstract 

Enhancing critical thinking skills for undergraduate students is important across the curriculum and between 
disciplines. We report on a method of improving critical thinking skills, which was studied through an Ethics 
and Science First-Year Seminar course. We used full cycle assessment over a three-year period to assess 
students’ development and to modify the course teaching and assignments with the goal of increasing student 
development of critical thinking skills. Data were obtained from student writing throughout the semester 
during each offering. Modest, but significant, overall gains of ~0.7 on a 4 point scale are reported between 
early and midterm assignments in the course using a seven trait assessment rubric. Key factors that contribute 
to the increase in critical thinking skills are identified including peer review, scaffolded assignments, and the 
use of a grading rubric for each assignment. 
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Abstract 

Enhancing critical thinking skills for undergraduate students is important across the 
curriculum and between disciplines. We report on a method of improving critical thinking 
skills, which was studied through an Ethics and Science First-Year Seminar course. We used 
full cycle assessment over a three-year period to assess students' development and to 
modify the course teaching and assignments with the goal of increasing student 
development of critical thinking skills. Data were obtained from student writing throughout 
the semester during each offering. Modest, but significant, overall gains of ~0.7 on a 4 
point scale are reported between early and midterm assignments in the course using a 
seven trait assessment rubric. Key factors that contribute to the increase in critical thinking 
skills are identified including peer review, scaffolded assignments, and the use of a grading 
rubric for each assignment. 

Keywords : Critical thinking, assessment, rubric, writing 


Introduction 

Understanding how students learn and the ways in which different pedagogical techniques 
affect that experience is the basis for promoting a rich and fruitful learning environment 
that serves everyone - individual students, faculty, educational institutions, and society at 
large - as these students graduate and move into the workforce. The premise that critical 
thinking skills are foundational to learning across many disciplines and that undergraduate 
curriculum should advance these skills for students is broadly supported by faculty across 
the nation. Improvement of critical thinking skills is in the national spotlight as institutions 
respond both to faculty concerns that students are failing to exhibit high level reasoning 
skills and to external pressures to document and assess student learning (American Council 
on Education 2004; Pusateri & Flurd 2006; U.S. Department of Education 2006). 

In this paper, we report on a full-cycle assessment of critical thinking in three years of 
teaching an ethics and science first-year seminar course. First we reflect on the definitions 
of critical thinking, and then we describe our methodology and assessment efforts. This is 
followed by a brief description of the ethics and science course including the pedagogy and 
the specific assignments that are assessed for critical thinking skills with a carefully 
developed rubric. Finally, we report our results over three years which allow us to identify 
key elements which improve students' critical thinking. 
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Improving Critical Thinking During College 

Research on critical thinking is based on a variety of assessment methods, as well as a 
variety of definitions of the term critical thinking. We find the following two definitions 
particularly useful. 

The Foundation for Critical Thinking uses the following description: 

Critical thinking is, in short, self-directed, self-disciplined, self-monitored, and self¬ 
corrective thinking. It requires rigorous standards of excellence and mindful 
command of their use. It entails effective communication and problem-solving 
abilities and a commitment to overcome our native egocentrism and sociocentrism. 
(Paul & Elder 2004, pg. 1) 

In her 1988 book on critical thinking, Joanne Kurfiss proposed the following: 

Critical thinking is a rational response to questions that cannot be answered 
definitively and for which all the relevant knowledge may not be available. It is 
defined here as an investigation whose purpose is to explore a situation, 
phenomenon, question or problem to arrive at a hypothesis or conclusion about it 
that integrates all available information and that can therefore be convincingly 
justified. In critical thinking, all assumptions are open to question, divergent views 
are aggressively sought, and the inquiry is not biased in favor of a particular 
outcome. (Kurfiss 1988, pg. 2) 

Both definitions of critical thinking focus on the ability to study an issue using problem 
solving techniques, which include assembling relevant data or information, being open to 
different points of view, and then forming conclusions based upon the evidence, rather than 
personal opinions. 

Just as there are a variety of definitions of critical thinking, assessment methodologies of 
student development of these skills range from student self-reporting of their abilities, to 
nationally available tests, to individual assessment rubrics developed by researchers. For 
example, several studies used students' self-report of their own critical thinking abilities to 
track improvements in student learning (e.g., Broadbear 2003, Tsui 1999). One concern 
with these studies is that students usually over-estimate their own ability to use skills they 
have recently been taught (Dunning et. al. 2004). Some studies use a subset of the 
Collegiate Assessment of Academic Proficiency (CAAP), a test developed by ACT to assess 
general education programs (http://www.act.org/caap, Pusateri & Hurd 2006). Other 
individual researchers (e.g. Bissel & Lemons 2005, Zeller &Tsaparlis 1997) have based their 
assessment of critical thinking on Bloom's Taxonomy of Educational Objectives (Bloom 
1984), in which questions of application, analysis, synthesis, and evaluation are ranked 
higher than questions of knowledge or comprehension. King and Kitchner (1994) have 
created a reflective judgment model of critical thinking, which defines seven levels of 
judgment. This is a developmental model; students need to reach the first level before they 
can reach the second, and so on. As students move through the levels, they become more 
able to identify different perspectives, and they learn to choose among them. These critical 
thinking skills can be applied to problem solving or to the development of an argument. 

Various aspects of attending college help students to increase their critical thinking ability 
(Astin 1993; McMillan 1987, Pascarella 1989, Terenzini et. al. 1995), but the reported 
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increases are modest. Most studies show that no individual experience or mode of 
instruction was crucial on its own, but that the aggregate of both in-class and out-of-class 
experiences changed students over a period of a year or more. In fact, effects are quite 
small, even when they are statistically significant. For example, Terenzini, et. al. found that 
the effects of college are smaller than the contributions of pre-college experiences, as 
measured by a pretest of critical thinking (Terenzini et. al. 1995). Some aspects of 
instruction do seem to help students develop their critical thinking. For example, giving 
students problems for which no single answer is correct, having students revise writing 
assignments, giving students the criteria for assessing critical thinking, and the use of 
scaffolded assignments have all been found to be helpful (Broadbear 2003, Butler, 
Stonewater & Kinney 2005, Astin 1993, Van Gelder 2005, Bean 2001, Lynch & Wolcott 
2001 ). 

Several recent studies have done more targeted assessments of undergraduate students' 
skills and have found that improvements in student abilities seem to be less than faculty 
might expect from their own anecdotal experiences. King and Kitchner (1994) found that 
students in their first year of college have a mean reflexive judgment score of 3.63 (on the 
scale of 1 to 7), and students in their senior year had a slightly improved average score of 
3.99 (King & Kitchner 1994, p. 161). Students with a score of 3 will try to find the "correct 
answer" to unstructured problems, ignoring contradictory evidence as they try to make 
things simpler. Students with a score of 4 can accept uncertainty, but have trouble 
distinguishing between opinion and evidence. 

Wolcott and associates have built on this work, adapting King and Kitchner's seven levels 
into five performance levels, labeled 0-4 (Wolcott & Gray 2003). At the lowest levels, 
students do not see their own biases. As students develop, they learn to see other people's 
point of view and move beyond the "perpetual analyzer" stage to the stage where they can 
use arguments to make practical decisions. Wolcott has found that 50% of college students 
in their first year are unaware of any assumptions they or others are making and insist that 
there be one correct answer to even the most open-ended questions (level 0). By the time 
students are seniors, a majority of them have moved from level 0 to level 1, where they can 
acknowledge other viewpoints, but usually only perceive evidence that supports their own 
opinion (Wolcott & Gray 2003). Again, this is a small improvement; very few students reach 
the highest levels of critical thinking in college. 

In order to formally measure critical thinking skills, The Center for Teaching, Learning, and 
Technology, the General Education Program and the Writing Program at Washington State 
University developed a critical thinking rubric to be used on written work 
(http://wsuctproject.wsu.edu). The rubric, developed in 1996, has the following seven 
traits: problem identification, establishing a position, consideration of other perspectives, 
consideration of context, providing evidence, examining assumptions, and drawing 
conclusions. (Brown 2004) Each of these traits was assessed on a scale of 1 to 6. Professors 
at Washington State University have found that using the rubric throughout a course helps 
students' critical thinking skills. In a 400-level course without the rubric, papers were rated 
an average of 1.9, while in the same course with the rubric the papers were rated an 
average of 3.5, a significant difference (Conlon & Kelly-Riley 2004, Kelly-Riley 2003). 

In addition, when faculty members at Washington State University assessed writing 
assignments, they found that many of them ask for a high level of critical thinking, and that 
students exhibited better critical thinking in their work when the faculty specifically asked 
for it (Brown 2004). 
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As we have seen above, several pedagogical techniques have been identified as methods for 
increasing student abilities in the area of critical thinking. In addition, a number of 
assessment tools have been developed to investigate the nature of student performance, 
each of which have common themes but different rating scales. The research on student 
development of critical thinking skills concludes that it is difficult to improve students' 
critical thinking skills, and thus modest gains are noteworthy. 


Settings and Subjects 

In this paper we present a full-cycle assessment of a seminar for first-year students at 
Miami University, an Ethics and Science course. In full-cycle assessment, the results of 
assessment are used to redesign assignments, which are then re-assessed the next time 
the course is taught. This has been found to be an effective method for course redesign 
(Butler, Stonewater, & Kinney 2005). We have used this iterative full-cycle assessment 
framework to address the question: "How do we improve students' critical thinking?" 

Miami University First-Year Seminar Classes 

Miami University began offering First-Year Seminar Classes in Fall 2004. The goal of these 
courses was not only to raise the intellectual climate for first-year students, but also to 
provide them with a low population, seminar-style course that fostered student-student 
interaction, student-teacher interaction, and student responsibility for learning. The courses 
have an enrollment cap of 20, and 9 to 14 are offered each semester. Faculty members 
were asked to submit proposals for these new courses and course topics were selected from 
these submissions. The seminars run 2-3 times, and then are replaced with new courses. 
The seminar discussed here is the Ethics and Science First-Year Seminar Course taught by 
Yarrison-Rice, which was offered in Fall 2004, Fall 2005, and finally in Spring 2007. 
Enriching first-year students' experiences is particularly important in terms of critical 
thinking, as Wolcott has reported that ~50% of entering college students are at the lowest 
development level (called "confused fact-finder" on Wolcott's) in their critical thinking skills 
as demonstrated through their writing (Wolcott & Gray 2003). 

General Structure 

The Ethics and Science Course begins with simple questions on the nature of science, and 
then considers topics such as scientific misconduct, ownership of data and intellectual 
property, human and animal experimentation, public health, conflicts of interest, and bias in 
research. A series of real-world current news articles, case studies, and short vignettes are 
used to explore these ideas and the ethical issues surrounding them. 

Specific topics are explored in the following manner. Students are given news articles or 
sections of the textbook (Elliot & Stern 1997) to read on a particular topic. An initial 
discussion is held (in some cases an expert is invited to come to the class and present 
information from that area), and case studies are used to elicit student opinion. Students, 
working in groups of four, then embark on fact-finding missions, and a debate or discussion 
on issues is held with different groups presenting different sides of the question and leading 
the discussion. The case study analysis is organized as follows. Students analyze case 
studies and news articles to learn about the details involved and to envision the different 
directions in which issues might further develop. They are asked to identify the following 
points in each situation that is considered. 

• What are the facts of the case? 

• What are the issues brought out in the study? 
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• Who is affected by the problem (may or may not be persons directly mentioned 
in stories)? What are their perspectives and values? 

• What professional knowledge would I need to know in order to make an informed 
decision about this topic? 

• What are possible actions that could be taken, and which are most important 
(prioritize)? Consider both short term and long-term actions. 

• What are the possible consequences of such actions? Consider ethical 
implications, practical constraints, and identify both positive and negative aspects 
of the course of action. 

Written assignments range from analyzing specific scenarios and taking a stance to defend 
to writing reflection papers on more global questions after a topic has been studied. 

This research project on critical thinking was born out of dissatisfaction with the students' 
critical analysis as demonstrated in their early written assignments in the first offering of 
the course. We realized that the case study oriented curriculum would allow us to track how 
students approach critical thinking. The analysis of case studies requires students to use the 
seven critical thinking traits of our own Washington State University-based rubric, and we 
thought a student's well-written assignment should also contain evidence of her/his use of 
these skills. These assessments would provide information that could be used to improve 
the second offering of the course. 

Assignments in Fall 2004 consisted primarily of two types of assignments: 1) reading and 
analyzing case studies using the "talking points" (bulleted above), and 2) a series of open- 
ended "reflection" questions for students to use to articulate a response to the issues we 
were considering in a given part of the course. The goal of the reflection questions was to 
lead students to synthesize the case study with their own personal values and ideas, as well 
as other people's perspectives, and to come to conclusions which they could then support 
with evidence from the cases. The three assignments described below were used for the 
critical thinking assessment. 

The first assignment was one in which students were asked to analyze real-life case studies 
of scientific misconduct investigations across the country. Students were broken into 
groups. Each group was responsible for one case study. Each person in the group was asked 
to take one additional question produced by the group and research it further. The new 
information on the case would add more depth of understanding of particular issues or 
provide a broader database from which to discuss and examine the issues of the scientific 
misconduct. 

The second assignment we assessed, the midterm, asked students to select their own 
individual case study and fully analyze it. Students did their own research using what they 
had learned earlier about online academic search engines in a session with a science 
librarian. Students were to find their own case studies or other science news stories which 
had a strong ethical component to them. Then they analyzed the cases using the talking 
points and wrote their midterm papers based upon these cases/science-based ethical 
issues. 

In the final assignment, the students are asked to address the following question: What 
ethical science issue do you feel strongly enough about to proactively work towards 
resolving? Students were asked to use different case studies from throughout the semester 
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to provide examples and to support their stances. These three assignments were quite 
open-ended and, in retrospect, seemed to leave first-year students somewhat unsure as to 
how to approach them, even by semester's end. Other assignments throughout the 
semester focused on more specific case studies and vignettes from our readings about 
particular topics, but these assignments also contained open-ended questions for students 
to reflect on the ideas presented in a very broad way. The premise was that students would 
feel freer to write about topics in their own way. Students were not presented with any 
information specific to critical thinking skills during the first offering of the course, but, as 
described later, they were introduced to critical thinking in subsequent course offerings. 


Methods 

The assessment of critical thinking in the papers written by the Ethics in Science students 
was carried out by Miami University Assessment Fellows. This group of a dozen faculty 
members including Taylor, representing a wide spectrum of academic disciplines, was 
constituted in the fall of 2003 for the purpose of enhancing the teaching and learning of 
critical thinking skills at Miami. Enhancing critical thinking is one of the four goals of Miami's 
general education curriculum, yet there has been little consensus on either the meaning of 
critical thinking or how to measure it. The goals of the Assessment Fellows were to create a 
definition of critical thinking that could be agreed upon by most disciplines, to develop 
means of assessing critical thinking, and to assist faculty in carrying out assessment 
projects. After yearlong discussions on assessing critical thinking, the Fellows agreed to 
adapt the Washington State University rubric for measuring seven primary traits of critical 
thinking. Because the research on critical thinking states that undergraduate students are 
not likely to develop through the full range of stages found in the Washington State rubric, 
the Miami University rubric was condensed to four stages of development. The rubric as 
modified by Taylor for use by the Assessment Fellows may be found in Appendix A (Blue, 
Taylor, & Yarrison-Rice 2006). The original Washington State Rubric only had descriptions 
for the highest and lowest points on the scale. Because the Fellows found this difficult to 
use, Taylor wrote descriptions for each of the intermediate stages, so that descriptions were 
complete for all four stages of each of the seven traits. 

The seven primary traits identified for assessment were 

1. Identifies and summarizes the problem/question at issue 

2. Identifies and presents the student's own perspective and position as it is 
important to the analysis of the issue 

3. Identifies and considers other salient perspectives and positions that are 
important to the analysis of the issue 

4. Identifies and assesses the key assumptions 

5. Identifies and assesses the quality of the supporting data/evidence and provides 
additional data/evidence related to the issue 

6. Identifies and considers the influence of context on the issue 

7. Identifies and assesses conclusions, implications, and consequences 
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The rubric describes four levels of performance for each of these traits: scant, minimally 
developed, moderately developed, and substantially developed, each of which is given a 
numerical rank, from 1 for "scant" through 4 for "substantially developed". The decision was 
made to assess only the first paper written, the midterm paper, and the final paper, 
because a high quality response on each of these assignments would require evidence of all 
seven critical thinking traits. 

Prior to assessing the Ethics and Science papers, the Assessment Fellows participated in 
several norming sessions lead by members of the English department, who were 
experienced in using rubrics to assess student writing, to make sure that all were 
interpreting the criteria and the rubric levels in the same way. Since then, the Fellows have 
carried out a number of assessment projects for the university. Inter-rater reliability has 
been good; typically the fellows' ratings differ by no more than one level 90% of the time. 

In accordance with our approved Institutional Review Board protocol, students in the class 
were informed of the research project and given the option of not having their work 
included. All students who did not opt-out signed an informed consent form. Thirteen out of 
twenty-one students agreed to participate the first year. (In Year 2, 13 of 19 students 
participated, and work from 11 of 20 students was assessed in the third year) Student 
names were removed from the papers and a number was assigned to each. These numbers 
were randomized so that it would not be obvious to the readers which papers were written 
early in the semester and which were from late in the semester. Records were maintained 
so that the researchers can tell which sets of three papers were written by the same 
student, but all data associating that set with a particular student was destroyed. Copies of 
all the papers are archived in the University Assessment Office. Each student paper was 
read and assessed by two Assessment Fellows using the modified Washington State rubric. 
Distribution of the papers was randomized so that each rater had some papers from each 
assignment, and so that it was unlikely a rater would have two papers from the same 
student. Averages were calculated for each trait. A frequency analysis of the number of 
times each of the four categories were used for each trait was also performed, which 
provides another lens through which to look at the data. 

We next present the results of the assessment of critical thinking development in the Ethics 
and Science class during each of the three years of this study. Using our modified 
Washington State rubric, papers were scored on the seven critical thinking traits on a scale 
of 1 to 4. Averages of each of the seven traits for all student scores were calculated, as 
were the changes in scores throughout the semester and between each different year of the 
course offering. In addition, a frequency analysis of the number of times a particular trait 
was scored with a particular score is provided in a series of tables. These results are 
reported in separate sections for each year the course was offered. Each year's results 
section is followed by a description of the modifications we made to the course after the 
assessments were conducted and analyzed. Lastly, a discussion of the conclusions we draw 
from the full-cycle assessment is provided. 


Results - Year 1 

The averages reported here are the average of all Fellows' ratings of all student papers on 
each trait (Table 1). The ratings on the early papers show students averaged between scant 
(1) and minimally developed (2) in all categories. Students were rated highest in their 
ability to identify the problem (average 1.8) and next highest at identifying their own 
perspective (average 1.5). The other critical thinking tasks had averages between 1.2 - 1.3. 
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However, even the highest score of 1.8 reflects that the students' ability to identify the 
problem is at best "minimally developed." 


Table 1: Average Scores, Year 1 (Average of all Fellows' ratings for each trait) 



Early Assignment 

Midterm 

Final 

Identify Problem 

1.8 

2.8 

2.7 

Student's Own Perspective 

1.5 

2.1 

2.9 

Other Perspectives 

1.3 

2.1 

1.8 

Assumptions 

1.2 

1.7 

1.6 

Using Evidence 

1.2 

1.7 

1.7 

Contexts 

1.3 

1.9 

1.8 

Conclusions 

1.3 

1.9 

1.9 


The students demonstrated considerable improvement between the early assignment and 
the midterm paper. Students were still best at identifying the problem (average 2.8, up 
from 1.8), and their next highest average ratings were in identifying both their own 
perspective and other perspectives (average 2.1, up from 1.5 and 1.3 respectively). The 
ratings on the other critical thinking tasks showed gains of 0.5 -0.6 from the early 
assignment to the midterm. For the midterm paper, the highest individual ratings approach 
a score of 3, which means "moderate" evidence. Overall, no improvement was found from 
the midterm paper to the final paper. Most of the averages stayed within 0.1 of what they 
had been at the midterm, with two exceptions. Students' average ability to identify their 
own perspective increased significantly, by 0.8 (from 2.1 to 2.9), while their average ability 
to identify other perspectives decreased from 2.1 to 1.8. 

One reason for the changes in averages of these two traits between the midterm and the 
final paper might be the change in the nature of the assignment. The midterm paper was a 
straightforward single case study analysis, while the assignment for the final paper, which 
asked students to choose an ethical issue they would be willing to work towards changing, 
may have prompted the students to think about their own perspective more than the 
perspectives of others. The overall lack of improvement between midterm and final also 
points to the differences in assignment, as a well-developed response for the final would 
require students to synthesize information from several case studies (rather than just one) 
which requires proficiency in higher-order skills than those needed for analyzing a single 
case study. 

Looking at the frequency data provides insight into the question of student improvement. 
The frequencies of the ratings given to student papers on the first assignment for Year 1 are 
shown in Figure la. These are not averaged; since two readers read each paper, both of 
those ratings are shown separately on the chart. The increases in individual students' scores 
are graphically exhibited as one views each trait's frequency distribution in vertical fashion 
from the early to midterm to final assignments. 

The frequency data for the first paper show that only one trait, identifying the problem, had 
many papers that received a 3 or 4. In fact, the rest of the traits were predominantly scored 
as 1 (or scant development) with a few papers receiving a 2. The frequency data for the 


https://doi.org/10.20429/ijsotl.2008.020110 




IJ-SoTL, Vol. 2 [ 2008 ], No. l, Art. 10 


Figure la: Year 1 - Early Assignn 
Frequencies of ratings 

25 - 

lent 

— 

20 - 

fPPPP 

— 

| ■ Scant 

1 ■ .Min. Dev 
| .Mod. Dev 

rK 

1 2 3 4 5 6 7 

Trait 

Figure lb: Year 1 - Miidterm 

Frequencies of ratings 

20 - 

— 

1 1 1 

■ Scant 

■ ■ .Min. Dev 

I .Mod. Dev 

(III 1 1, 1 

| ■ .Sub. Dev 

h 

1 2 3 4 5 6 7 

Trait 

Figure lc: Year 1 - Final 

Frequencies of ratings 

20 - 


In i htrfr 

Scant 

■ .Min. Dev 

.Mod. Dev 

|l ■ .Sub. Dev 

£33 


1 2 3 4 5 6 7 

Trait 


https://doi.org/10.2042 9/ij sotl.2008.02011o 


9 





























































































Full-Cycle Assessment of Critical Thinking in an Ethics and Science Course 


midterm paper shows a strong shift upwards, as shown in Figure lb. The Fellows gave far 
fewer ratings of 1 and far more ratings of 3 on the midterm paper than on the first paper. 
The rating most often given is no longer a 1, but a 2. The upward trend from the midterm is 
continued into the final in terms of the frequency distribution. Recall that the averages for 
five of the seven traits were very similar, the exceptions being traits 2 (student perspective 
which increased by 0.8) and 3 (other perspectives which dropped by 0.3). Flowever from 
the frequency data, one can see that there are many more ratings of 3 given than there 
were on the midterm paper and a few traits received scores of 4. One can also see that 
some students reverted to scores of 1 on some traits. Thus, while the average scores 
remained about the same, individual scores did not. Again, this may be the result of the 
nature of the assignment for the final paper. Some students were able to use the open- 
ended nature of the assignment to showcase their critical thinking skills, while others 
apparently did not interpret the assignment as requiring anything other than their own 
perspectives. 


Course Modifications for Year 2 

The first offering of the course did not focus on the development of critical thinking in terms 
of instruction. As a result of the assessment for the three different assignments in the first 
year, the authors decided on several modifications for the second offering of the Ethics and 
Science course. We decided to openly discuss critical thinking with the students and to 
make development of such skills a parallel goal for the course. Because even at semester's 
end in the first course offering students seemed to exhibit mostly lower-level critical 
thinking skills, we decided that we needed to change the next course offering to include 
multifaceted approaches to the development of critical thinking including: 

• assignments designed specifically to teach the ideas behind critical thinking 

• assignments which broke down the critical thinking skills into smaller increments 

• assignments which built from the lower level critical thinking skills up to more 

difficult skills -- or scaffolded assignments 

• some assignments interspersed through the semester which brought all the skills 
together 

• use of a grading rubric which outlined the particular critical thinking skills we were 
targeting in each assignment 

We also decided to provide the students with the grading rubric in advance, concurrent with 
each assignment, so that students could see what was expected. Grading rubrics (to be 
differentiated from critical thinking assessment rubrics) had not been used in the first 
offering of the course. We designed the grading rubrics to contain the different aspects of 
critical thinking and give specific point values for various skills, which varied throughout the 
semester as students became more sophisticated in their application of critical thinking 
skills. Students were also given a handout that elaborated on the meanings of each of the 
seven critical thinking traits providing specific examples related to course materials. 

Assignments in Fall 2005 (Year 2) were based on many of the same case studies as the 
previous year, although some readings changed as the news brought different ethical issues 
to the forefront. Our research group met several times throughout the semester to select 
different types of scaffolded assignments that could be given and to work out a grading 
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rubric. The revised assignments followed the above bulleted guidelines within the curricular 
needs of the course. 

The first writing assignment in the second year focused on the ethics of doctors being paid 
by investment companies to answer questions about research or the general state of a 
current medical topic. Students were asked to apply the seven critical thinking traits to this 
case study in an essay format. The second (midterm) assignment was basically the same as 
the midterm in the first year. Students were asked to choose their own real-life ethical 
dilemma and study it using news articles and/or websites with pertinent information on the 
topic, so that they could analyze the dilemma. The major difference in the assignment was 
that the grading scheme for the midterm essays was provided in advance. 

The assignment for the 2005 final paper was more focused than the previous year's final 
assignment. Students were asked to write a 4-6 page paper on one question - Should 
society's perceived need for information ever override individual rights? They were told to 
use specific case studies and classroom discussions to describe their views and support their 
opinions. Students had to integrate evidence from several sources rather than concentrating 
on a single case study. The grading scheme for the final essay is found in Appendix B. The 
grading rubric for the midterm was similar to that used for the final essay, but with the last 
few traits weighted less heavily. 

Part of our goal with the second year's offering of the seminar was to move students 
through different assignments, each of which would highlight different critical thinking skills. 
In addition, as the semester progressed the grading rubrics would then be weighted more 
heavily toward the more difficult levels, as students mastered the lower level skills. We use 
the term "scaffolded assignments" to describe this progression, and give more details on 
some of these assignments below. 

For example, early in the semester students were asked to simply identify the main 
dilemma a case study presented and to describe what the issues were and who was affected 
by the problem. Then they were asked to present their own perspective on the issue. At 
every step of the way, students were expected to document their findings with evidence 
from outside references. Students were then asked to consider other affected parties' 
positions on an issue. This forced them to think outside their normal comfort zone and to 
put themselves in other people's places. Students' ability to do this varied significantly from 
person to person. The use of an in-class formal debate for two topics forced them to 
consider both sides of different issues and really appealed to the students. In fact, one 
assignment was key to their development of this ability. They were asked to write a pro¬ 
statement and a con- statement about the use of animals in research, and to write them in 
such a way that a reader could not decide which opinion was actually theirs. Students did 
surprisingly well at making a strong argument on both sides of the issue. 


Results - Year 2 

The Assessment Fellows used the same rubric to rate the essays written to fulfill the 
modified assignments and teaching strategies used in Year 2. Again, only the first paper, 
midterm, and final paper were used for the critical thinking assessment, because they were 
comprehensive enough to require the use of all seven primary traits on the critical thinking 
assessment rubric. The evaluations of student critical thinking from Year 1 and Year 2 were 
very similar, as presented in Table 2. We see a gain in student learning of critical thinking 
skills between the early and midterm assignments, and then a plateau to the final 
assignment. The gain in average scores for Year 2 were highest in the first three traits. In 
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trait #1, identifying the problem the gain was around 1 (from an average score of 1.5 to 
2.5) which was the same gain seen in Year 1, with the students' ability to relay their own 
perspective gaining 0.7 (from 1.8 to 2.5), and for identifying other people's perspectives 
students gained 0.6 (from 15 to 2.1). In the other four traits, student gains varied between 
0.3 and 0.5 in Year 2. In the final paper, students' scores either stayed the same or dropped 
slightly across all traits (which is why they are not shown on this table). This contrasts to 
the final paper in Year 1 where the students' own perspective showed a large gain. 


Table 2: Average Scores, Years 1 and 2 (Average of all Fellows' ratings for each trait) 



Early 

Assignment 
Year 1 

Early 

Assignment 
Year 2 

Midterm 
Assignment 
Year 1 

Midterm 
Assignment 
Year 2 

Identify Problem 

1.8 

1.5 

2.8 

2.5 

Student's Own Perspective 

1.5 

1.8 

2.1 

2.5 

Other perspectives 

1.3 

1.5 

2.1 

2.1 

Assumptions 

1.2 

1.2 

1.7 

1.7 

Using Evidence 

1.2 

1.4 

1.7 

1.8 

Contexts 

1.3 

1.5 

1.9 

1.8 

Conclusions 

1.3 

1.5 

1.9 

1.9 


The final paper in Year 2 was required integration of several case studies to answer the 
question in a defined topic, so while student' critical thinking skills may be improving for 
single topic analysis, when asked to address a broader question drawing evidence from 
several cases, students critical thinking scores were rather level. The frequency data for this 
paper show that individual scores increased over Year 1 with more ratings of 2 and 3. (See 
Figure 2.), However, the numerical averages were not much different between years 
indicating that individual student learners exhibit some higher gain in Year 2 than 1, but 
that the class as a whole experienced a similar overall gain. An individual student's increase 
in performance might be attributed to our clarifying the assignments and using scaffolding 
assignments. The results of the frequency distribution comparing the midterm assignment 
with the final assignment are very similar to Year 1 results with more spread seen in the 
individual scores. 


Course Modifications for Year 3 

Before the third year this course was offered, we reviewed the course materials and 
assignments in light of the results of our critical thinking assessment. Because in the second 
year rubrics assisted both the faculty member and the students as they considered, wrote, 
and then graded assignments, we wanted to continue their use in the course. However, 
since student writing seemed to be more stilted when the rubrics were used (often 
assignments became as a seeming check-list for the rubrics), we decided to provide a less 
detailed rubric which identified the major requirements for an assignment, but did not order 
them particularly or assign point values in advance. Our thesis was that the positive effect 
of seeing the different critical thinking requirements would remind students about how to 
approach a particular assignment without losing their individuality. Several more scaffolded 
assignments were designed for the course as well. 

Another conclusion we drew from the first two times the course was run was that the use of 
open topics, where students choose topics themselves, for midterms and finals seemed to 
negatively affect students' critical thinking and analysis of case studies. In particular, 
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students tended to pick topics for which they had strong opinions, and this had the effect of 
clouding their critical thinking in terms of being able to analyze dispassionately other points 
of view and to find good evidence to support opposing ideas. Thus, we decided to limit 
students' topic selection and provided a series of 5-7 topics for students to choose from for 
their midterms and finals in order to let them analyze and write more balanced papers. In 


Figure 2a: Year 2 - Early Assignment 
Frequencies of ratings 
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Figure 2c: Year 2 - Final 
Frequencies of ratings 
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order to have uniformity from year two to three, we kept the same final exam question for 
the students to address with the small modification that students could consider the rights 
of small groups of people versus larger populations or national/international good, rather 
than just the rights of the individual, if their particular topic merited such an approach. 

In the third year, students were also required to have a peer review of their midterm and 
final papers, and were given the opportunity to turn in an outline for instructor review 
before their final papers were due. In previous years, peer reviews were suggested, but not 
required, for all the papers. 


Results - Year 3 

The third time the Ethics and Science course was offered was in the spring of 2007. 

Students in this seminar thus had a full semester of undergraduate education before joining 
this course; in contrast, the first two times the course was conducted were in the fall 
semester when students first came to the university. We were interested in how this 
semester's worth of preparation and experience would affect the critical thinking baseline of 
our students, and indeed found that students began the second semester of college at a 
higher critical thinking level than in the first semester. This is evidenced in the averages of 
the first assignment of the course. (See Table 3.) In Year 3, the average assignment score 
was higher by approximately 0.4 (calculated from the difference between Year 3 and the 
average of Year 1 and Year 2 scores together). Thus, it is more instructive to look at 
changes in average scores for the student gains during the semester, rather than the 
absolute scores they received. (See Table 4.) 

In the first assignment of Year 3, students had their highest averages for the first three 
traits: identifying the problem (2.0), describing their own perspective (2.2) and recognizing 
other people's perspective (1.9). The first two traits were high for all three years in the first 
paper. The gains from the early assignment to the midterm are strong in three areas: in 
identifying the problem (0.8 gain), in using evidence (1.0 gain), and in context (where a 0.8 
gain was observed). The first gain was also seen in Years 1 and 2; however, the gain in 
using evidence to support conclusions and discussions and identifying contexts was much 
stronger in Year 3 than previous years (Year 1 changes were 0.5 and 0.6 and Year 2 
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changes were 0.4 and 0.3 respectively). The other traits had gains of 0.6 to 0.7 for Year 3. 
Once again with the midterm to final assignment, the Year 3 averages either did not change 
or were slightly lower. 


Table 3: Average Scores, Year 3 (Average of all Fellows' ratings for each trait) 



Early Assignment 

Midterm 

Final 

Identify Problem 

2.0 

2.8 

2.5 

Student's Own Perspective 

2.2 

2.9 

2.6 

Other Perspectives 

1.9 

2.5 

2.2 

Assumptions 

1.5 

2.1 

2.2 

Using Evidence 

1.5 

2.5 

2.2 

Contexts 

1.6 

2.4 

2.0 

Conclusions 

1.8 

2.5 

2.2 


Table 4: Differences Between Averages on Midterm and Early Assignments 


(Average for each trait on midterm - average for each trait on early assignment) 



Year 1 

Year 2 

Year 3 

Identify Problem 

1.0 

1.0 

0.8 

Student's Own Perspective 

0.6 

0.7 

0.6 

Other Perspectives 

0.8 

0.6 

0.6 

Assumptions 

0.5 

0.5 

0.6 

Using Evidence 

0.5 

0.4 

1.0 

Contexts 

0.6 

0.3 

0.8 

Conclusions 

0.6 

0.4 

0.6 


The frequency distributions for Year 3 are shown in Figure 3 for the three assignments. The 
distributions are quite striking with a strong shift toward higher scores for individual 
students. Trait 1 (identifying the problem) and Trait 2 (student's own perspective) show 
many scores of 3 and some of 4 on the midterm assignment, with a drop in development 
shown in the final paper, which has a more even distribution across scores. The last 5 traits 
have strong gains from the early assignment to midterm and then also show that several 
individual students shift up again (to scores of 4 or substantially developed), while several 
individual students drop down in the last assignment. In the areas of other perspectives and 
use of evidence the final assignment students generally shift downward in their scores. 
However, the scores of 4 represent good gains for the individual students as they move 
through the course. In Years 1 and 2, very few students demonstrated substantial 
development of critical thinking skills and only in 3 or 4 of the 7 traits. Year 3 data have 
some individuals with substantial development of critical thinking skills across all 7 traits. 


Discussion 

We consider the results from all three years collectively in order to draw conclusions about 
pedagogical techniques that are effective in advancing students' critical thinking skills. In 
this study, we have seen a significant improvement in students' critical thinking from the 
early to the midterm assignments. Even in Year 3, where students began with higher scores 
on their first papers, improvements were seen at midterm. In the first year we attribute that 
gain to the fact that the early paper was rewritten after feedback from the instructor and 
peers. In the second year, we attribute the gain to providing the students with rubrics that 
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helped them apply critical thinking skills to a particular assignment, some use of peer 
review, and scaffolded assignments targeting specific critical thinking skills such as 
identifying contexts and assumptions. In the third year, we used the modifications from 
Year 2 with some additions like slightly modified assignments and rubrics (detailed above); 
in addition peer reviews were required and students were encouraged to outline their papers 
before writing first drafts. More specific discussions were held in Year 3 on how critical 
thinking assisted in analyzing ethical issues and this was reflected in the larger overall gains 
between the early and midterm assignments in Year 3 particularly with Traits 5 and 6. 

In Years 2 and 3, the instructor found the rubrics made the papers easier to grade, made 
the grading less subjective and, therefore, made the grades easier to explain to the 
students. At the end of the second year, a brief survey was administered to the students to 
determine their perceptions of the usefulness of the rubrics. Students were given four 
positively worded statements related to the rubrics and were asked to respond on a six- 
point scale from strongly agree (1) to strongly disagree (6). 


Figure 3a: Year 3 - Early Assignment 
Frequencies of ratings 
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Figure 3b: Year 3 - Midterm 
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Figure 3c: Year 3 - Final 
Frequencies of ratings 






The results of the survey are shown in Table 5. Clearly, the students found the rubrics 
helpful. Students strongly agreed with the usefulness of the rubric in writing (1.12 out of 6) 
and organizing their papers (1.24 out of 6). Standard deviations for these two questions 
were 0.33 and 0.44 indicating strong agreement among students. Students only somewhat 
agreed that the rubrics assisted them with bringing critical thinking into their papers (2.35 
out of 6) with a high standard deviation (1.37). We believe the difference between the 
student responses on the questions about general usefulness (Q1 & Q2) and usefulness for 
incorporating critical thinking into their writing (Q4) results from a lack of understanding of 
what critical thinking means. Interestingly, this misunderstanding persists even after a 
semester of instruction aimed specifically at improving critical thinking skills. Students were 
surveyed orally about the use of rubrics after the third year and they responded positively 
again about their usefulness in paper writing. The student consensus in both years was that 
rubrics should continue to be used in future offerings of the course. 


https://doi.org/10.2042 9/ij sotl.2008.02011o 


17 






























































Full-Cycle Assessment of Critical Thinking in an Ethics and Science Course 


Table 5: Student Evaluation of the Rubric, Year 2 



(1) Strongly Agree to 
(6) Strongly Disagree 

1. Flaving a rubric before I wrote papers was useful 

1.12 ± 0.33 

2. Flaving a rubric before I wrote papers helped me organize my 
writing 

1.24 ± 0.44 

3. Flaving a rubric helped me understand my grade 

1.88 ± 1.27 

4. Rubrics helped me incorporate critical thinking into my writing 

2.35 ± 1.37 


From Year 1 to Year 2, no significant difference between the midterm critical thinking scores 
was seen. This points to the difficulties in affecting changes in critical thinking as discussed 
in the Introduction. Flowever, when comparing the reported scores from Years 1 and 2 for 
each trait with the scores from Year 3, we see improvements in two traits in the midterm 
and two in the final papers. And frequency data shows that some students experienced a 
significant improvement over the semester, as scores of 4 were observed more frequently in 
the final paper in all seven traits for the Year 3 data compared to Years 1 and 2 where 
scores of 4 were seen in only four and three traits respectively. 

In all three years, the average critical thinking scores did not improve from the midterm to 
the final. The research of King and Kitchner (1994), Terenzini et. al. (1995), and Wolcott 
and Gray (2003) found only small changes in students' critical thinking skills over their 
entire college career, and so in our case the plateauing of student scores could be due to 
having reached the maximum amount of change that is possible to make in a time period as 
short as a semester. In addition, the final exam required a more global application of critical 
thinking skills than the midterm, as students had to integrate a series of case studies 
together to answer the final exam question; thus, the final assignment was more 
demanding than the midterm. This increased expectation in the complexity of the 
assignment may have masked the students' gains in critical thinking ability. 


Conclusions 

In conclusion, we find that an Ethics and Science seminar provides a good platform for 
studying students' development of critical thinking skills, as the course uses case studies, 
and thus provides a wide variety of issues for students to analyze using these skills. We 
have observed that students have a significant increase in their ability to identify the 
problem, and provide their own perspective over the first half of the semester in all three 
years of this study. Changes in students' critical thinking exhibited increases for all seven 
traits from the first assignment to the midterm of ~0.7 on a full scale of 4, with the 
particular traits of using evidence as support and understanding contexts showing larger 
gains in the third year. The students maintained, or increased, their critical thinking scores 
through the end of the semester as evidenced by the average scores of the final paper 
staying nearly the same and the larger number of students with scores of 4 (highest rating, 
"substantially developed" trait or skill) in the final assignment. This improvement in the 
frequency data was particularly noteworthy in the third offering of the course. 

The results from the three offerings of the Ethics and Science seminar are encouraging. 
Students are indeed developing their critical thinking skills over the course of the semester. 
A review of other studies indicates that the magnitude of improvement we record, while 
modest numerically, is significant. For instance, King and Kitchner found that students 
typically move one point on their five-point scale or ~20% between first year and the end of 
college (King & Kitchner 1994). Flere we observe a comparable increase of 16% over a 
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semester for first-year students, with a number of students moving upward to critical 
thinking score of 4 across the seven different traits in their final papers in the Year 3 
offering of the course. 

Factors that most likely contributed to these gains include: 

• carrying out discussions specifically on the analysis of case studies including specific 
aspects of critical thinking 

• using scaffolded assignments which focused on particular critical thinking traits in 
order of complexity 

• providing students with a rubric for assignments 

• providing students with a short list of possible cases and issues for major papers, 
rather then leaving the choices for students fully open 

• having students peer review rough drafts of major assignments and having them 
rewrite the drafts for final submission 

Finally, we conclude that enhancing students' critical thinking skills requires the instructor to 
use a series of pedagogical techniques: from the choice of ethical issues for students to 
analyze, to careful crafting of assignments and classroom activities, to the use of peer 
reviews and rubrics. No single approach guarantees success. Flowever, when an instructor 
incorporates these ideas together into a course, we find that it is possible to promote the 
development of critical thinking skills and that students demonstrate a significant gain in 
their abilities. Thus, the use of full-cycle assessment in a course can provide faculty with the 
data necessary for identifying which teaching methods are most effective and for coming to 
a better understanding of student development of critical thinking skills. 
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Appendix A 

Miami University Adaptation of the Washington State University Critical Thinking Rubric 


(1- scant) 

(2 - minimal) 

(3 - Moderate) 

(4 - Substantial) 

1) Identifies and summarizes the 

problem/question at issue (and/or the source's position). 


Scant 

Minimally Developed 

Moderately Developed 

Substantially Developed 

-Does not identify and 
summarize the problem, is 
confused or identifies a 
different and inappropriate 
problem. 

-Does not identify or is 
confused by the issue, or 
represents the issue 
inaccurately. 

-Identifies the main problem or 
question but does not recognize 
subsidiary or implicit aspects of 
the problem. 

-Demonstrates a basic 
understanding of the 
assignment. 

-Identifies the main problem or 
question and some but not all 
of the complexities or nuances 
associated with the problem. 
-Demonstrates a good 
understanding of the 
assignment and related course 
material. 

-Identifies the main problem 
and subsidiary, embedded, or 
implicit aspects of the problem, 
and identifies them clearly, 
addressing their relationships to 
each other. 

-Identifies not only the basics 
of the issue, but recognizes 
nuances of the issue. 

2) Identifies and presents the STUDENT'S OWN perspective and position as it is important to the analysis of the issue 

Scant 

Minimally Developed 

Moderately Developed 

Substantially Developed 

-Addresses a single source or 
view of the argument and fails 
to clarify the established or 
presented position relative to 
one's own. Fails to establish 
other critical distinctions. 

-Fails to acknowledge the 
existence of valid counter 
arguments. 

-Identifies one's own position 
on the issue relative to other 
positions, but does not provide 
supporting evidence for the 
position. 

-Fails to acknowledge the 
possible validity of other 
positions 

-Identifies, appropriately, one’s 
own position on the issue, 
drawing support from 
experience and information 
from course materials. 
-Recognizes that there are 
other valid points of view. 

-Identifies, appropriately, one's 
own position on the issue, 
drawing support from 
experience and information not 
available from assigned sources 
-Recognizes counterarguments 
that might be made and 
responds to them. 

3) Identifies and considers OTFIER salient perspectives and positions that are important to the ana 

lysis of the issue. 

Scant 

Minimaiiy Developed 

Moderately Developed 

Substantially Developed 

Deals only with a single 
perspective and fails to discuss 
other possible perspectives, 
especially those salient to the 
issue. 

Acknowledges that other 
perspectives exist, but fails to 
adequately present the case for 
these perspectives. 

Appropriately addresses 
multiple perspectives, but omits 
at least one important 
perspective. 

Addresses perspectives noted 
previously and additional 
diverse perspectives drawn 
from outside information. 

4) Identifies and assesses the key assumptions. 

Scant 

Minimally Developed 

Moderately Developed 

Substantially Developed 

Does not surface the 
assumptions and ethical issues 
that underlie the issue, or does 
so superficially. 

Identifies some but not all of 
the assumptions that have 
been made in their analysis. 

Only superficially considers the 
validity of those assumptions. 

Identifies the assumptions 
made in the analysis and 
considers their validity, yet fails 
to surface important ethical 
issues. 

Identifies and questions the 
validity of the assumptions and 
addresses the ethical 
dimensions that underlie the 
issue. 
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(1- scant) _ (2 - minimal) _ (3 - Moderate) _ (4 - Substantial) 


5) Identifies and assesses the quality of supporting data/evidence and provides additional data/evidence related to the issue. 

Scant 

Minimally Developed 

Moderately Developed 

Substantially Developed 

-Merely repeats information 
provided, taking it as truth, or 
denies evidence without 
adequate justification. Confuses 
associations and correlations 
with cause and effect. 

-Does not distinguish between 
fact, opinion, and value 
judgments. 

-Does not identify sources or 
uses inappropriate sources. 

-Provides significant supporting 
evidence only the student's 
own perspective. Evidence for 
other perspectives is minimal. 
-Does not examine the 
evidence for bias or 
incompleteness. 

-Does not recognize value 
judgments. 

-Identifies sources but has 
some questionable sources. 

-Provides significant evidence 
for multiple perspectives. 
Questions accuracy and 
completeness. Of some 
evidence, but not all. 

-May have some problems with 
cause and effect. 

-May fail to properly identify 
some opinions and value 
judgments. 

-Uses and cites appropriate 
sources. 

-Provides significant evidence 
for multiple perspectives. 
Examines the evidence and 
source of evidence; questions 
its accuracy, precision, 
relevance, completeness. 
-Correctly identifies cause and 
effect. 

-Clearly distinguishes between 
fact, opinion, & acknowledges 
value judgments. 

6) Identifies and considers the influence of the context on the issue. 

Scant 

Minimally Developed 

Moderately Developed 

Substantially Developed 

-Discusses the problem from 
the perspective of a single 
discipline. 

-Does not present the problem 
as having connections to other 
contexts-cultural, political, etc. 

Recognizes the importance of 
issues such as political and 
economic feasibility their 
question, but does not discuss 
their impact. 

Analyzes the issue considering 
relevant contexts, but fails to 
consider one important context. 

-Analyzes the issue with a clear 
sense of scope and context, 
including an assessment of the 
audience of the analysis. 

-Fully considers pertinent 
contexts such as political, 
economic, and social. 

7) Identifies and assesses conclusions, implications and conseq 

luences 

Scant 

Minimally Developed 

Moderately Developed 

Substantially Developed 

Fails to identify conclusions, 
implications, and consequences 
of the issue or the key 
relationships between the other 
elements of the problem, such 
as context, implications, 
assumptions, or data and 
evidence. 

-Fails to reflect upon own work. 

-Draws incomplete conclusions 
or considers only some of the 
consequences of the 
conclusions. 

-Fails to reconsider 
assumptions identified earlier. 
-Fails to reflect upon own work. 

-Draws appropriate conclusions 
from evidence/data. Identifies 
and discusses some, but not 
all, consequences of these 
conclusions. 

-Fails to consider the limitations 
of their own work. 

-Identifies some directions for 
further inquiry. 

-Identifies and discusses 
conclusions, implications, and 
consequences considering 
context, assumptions, data, 
and evidence. 

-Objectively reflects upon their 
own assertions. Acknowledges 
the value judgments on which 
their position is based. 

-Identifies appropriate 
directions for further inquiry. 
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Appendix B: Grading Scheme for 7 Critical Thinking Traits in a Case Study 


Talking Pt. Category 

# Pts. Possible 

Worth total of 100 pts. 

Extra Credit Poss. 6 pts. 

Main Ideas Expected 


1. Identify & Summarize Problem or Question at Issue 

worth 10 pts. 

Ethics of doctors being paid by investment companies to answer questions 

about research or general state of a current medical topic 

(6) 

Info for $ -lots of income to doctor 

(2) 

Confidentiality of medical procedures, etc. 

(2) 

Insider info -- few people/investors with more information (2) 

Regulation of such conduct 

(2) 

Other ideas 

(2) 

2. Student's perspective and position -- 

worth 20 pts. 

Provide perspective 

(10) 

Provide reasons why 

(10) 

3. Other important perspectives and positions -- 

worth 25 pts. 

Provide 1 other perspective 

(5) 

Provide 1 other set of reasons 

(5) 

Provide 2 nd other perspective 

(5) 

Provide 2 nd set of reasons 

(5) 

Provide more perspectives 

(5) 

4. Identify key assumptions -- 

worth 10 pts. 

Assumptions you make -- what you think is important 

(5) 

Assumptions others might make in other perspectives 

- #1 (2.5) 


- #2 (2.5) _ 

5. Quality of supporting evidence 

worth 10 pts. 

Your point of view 

(5) 

Others' points of view 

(2.5) 

Additional evidence you found (other references) 

(2.5) 

6. Influence of context on issue -- where does each person/attitude come from? 


worth 10 pts. 

Your context 

(5) 

Other perspective context #1 

(2.5) 

Other perspective context #2 

(2.5) 

7. Conclusion -- identifies implications and consequences 

worth 15 pts. 

Possible outcomes 

(5) 

Implications -practical drawbacks or positives of outcome 

(5) 

Consequences of particular outcomes for particular parties 

(5) 
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